The Uncommon Denominator: A Proposal for Consistent Reporting of Pronoun Resolution Results

Precision computes how well a technique did what it was designed to do, and is not at issue here. Recall is in tended as a more general performance measure, yet R scores are difficult to interpret due in part to varying methods of calculating T. T includes only the pronouns that were included in the s tudy rather than all p ronouns in the data set. But since different studies consider different sorts of pronouns to be in scope, R scores f rom different studies are difficult to compare. Also, since the pronouns in scope for a s tudy might represent a large or small percentage of the pronouns in the corpus, R reveals little about a technique's utility for the general problem of p ronoun resolution. This paper proposes a new report ing format and a new performance measure to supplement R and P. Pronoun resolution studies differ in m an y respects, such as the method of calculating C and the under lying semantic assumptions, and this proposal does not address ways to make the studies themselves more consistent (for discussion of these issues, see Walker 1989, van Deemter and Kibble 1999, Mitkov 2000). Instead, we propose a report ing format that clarifies the details of a s tudy 's test data (especially those details that tend to differ be tween studies) and explicitly derives the numbers used to compute performance measures.